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TABLE 2 

PROPERTIES OF EXTREMELY HIGH REDSHIFT GALAXY CANDIDATES 

Galaxy x y a S /a (3000) /a (4500) Ja (6060) /a (8140) (12. OOP) (16. OOP) 

A 115 481 12:36:56.3537 62:13:21.759 -P.PP3(P.PP3) P.PPP{P.PP2) P.PPl(P.PPl) P.PP3(P.PP2) P.15P(P.117) P.192{P.192) 1 

B 528 454 12:36:47.9797 62:12:51.286 P.PP1(P.PP4) P.PPP{P.PP2) -P.PP2(P.PP2) P.PPP(P.PP3) P.28P(P.14P) P.35P{P.23P) 1 

C 647 112 12:36:48.6375 62:11:53. 77P -P.PP1(P.PP3) P.PPP{P.PP2) P.PPP(P.PPl) P.PP2(P.PP2) P.1P1(P.141) -P.131(P.232) 1 

D 924 373 12:36:4P.4616 62:12:14. 0P4 -P.PP7(P.PP4) -P.PP2(P.PP2) P.PPP(P.PPl) P.PPP(P.PP2) P.444(P.156) 0.178(0.256) 1 

E 281 453 12:36:53.1447 62:13:07.014 P.PP6(P.PP4) -P.PP1(P.PP2) P.PP2(P.PP1) P.PPP(P.PP2) P.151(P.151) P.247(P.247) 1 



NOTE — Table lists galaxy, pixel values in the infrared imago x and y, J2PPP coordinates Q and S, fluxes /a(3PPP), /a(45PP), /a(6P6P), /a(814P), /a(12, 000), /;^(1 
to flux in K band, with flux uncertainties in parentheses), and magnitude AB{22, PPP). 
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ABSTRACT 



We apply the Lyman absorption signature to search for galaxies at redshifts z ~ 6 — 
17 using optical and infrared images of the Hubble Deep Field. The infrared images are 
sensitive to a point source 5a detection threshold of AB(22, 000) = 23.8, which adopting 
plausible assumptions to relate rest-frame ultraviolet flux densities to unobscured star 
formation rates is easily sufficient to detect the star formation rates expected for massive 
elliptical galaxy formation to quite high redshifts. For qq = 0.5, the infrared images are 
sensitive to an unobscured star formation rate of M = 100 Mq yr~^ to redshifts 
as large as z = 17, and for qq = 0, the infrared images are sensitive to an unobscured 
star formation rate of M = 300 Mq yr~^ to redshifts as large as z = 14. The 
primary result of the analysis is that only one extremely high redshift galaxy candidate 
is identified at the 5cr level of significance (and four at the 4cr level). This implies a 
strict upper limit to the surface density of extremely high redshift galaxies of < 1.5 
arcmin"^ to a limiting magnitude threshold ^.6(22,000) = 23.8. This also implies a 
strict upper limit to the volume density of extremely high redshift galaxies if (and only 
if) such galaxies are not highly obscured by dust. 



Subject headings: galaxies: evolution 
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1. INTRODUCTION 

A standard paradigm is that massive elliptical galaxies form most or all of their stars in a 
single rapid burst at early epochs. According to this scenario, early elliptical galaxies undergo a 
phase of intense ultraviolet luminosity, as the star formation rates required to produce 10^^ stars 
in under « 10^ yr exceed 100 Mq yr~^. Recent progress in identifying galaxies at high redshifts has 
opened up the possibility of directly observing this epoch of massive elliptical galaxy formation. In 
particular, various analyses of the Hubble Deep Field (HDF) images obtained by the Hubble Space 
Telescope (HST) have identified galaxies with spectroscopically determined redshifts as large as 
z ^ 4 (Steidel et al. 1996; Lowenthal et al. 1997) and photometrically estimated redshifts as large 
as z !=a 6 (Lanzetta, Yahil, & Fernandez-Soto 1996; Lanzetta, Fernandez-Soto, & Yahil 1997). A 
primary result of these analyses is that the unobscured star formation rates of high-redshift galaxies 
inferred directly from the observed ultraviolet continua are surprisingly modest — typically less than 
~ 10 Mq yr~^ and always well less than the ~ 100 Mq yr^^ or more expected for massive elliptical 
galaxy formation. Various proposals have been put forward to explain the apparent discrepancy 
between the expected and observed star formation rates, including the possibilities that (1) massive 
elliptical galaxies form at lower redshifts, say z < 2.5 (Madau et al. 1996), (2) massive elliptical 
galaxies form at redshifts z ^ 3 but are obscured by dust (Mcurer et al. 1997), and (3) massive 
elliptical galaxies form at higher redshifts, say z > 6 (Maoz 1997). 

The possibility that massive elliptical galaxies form at redshifts z > 6 is accessible to direct 
investigation by means of existing observations of the HDF. By combining images obtained at optical 
wavelengths by HST (Williams et al. 1996) with images obtained at infrared wavelengths by the Kitt 
Peak National Observatory (KPNO) 4 m telescope (Dickinson et al. 1998, in preparation), galaxies 
may in principle be detected at redshifts as large as z ^ 17, beyond which Lya is redshifted past 
the response of the K-hand filter. Such galaxies almost certainly must exhibit the same Lyman 
absorption signature characteristic of other high-redshift galaxies — namely complete absorption 
of flux below the Lyman limit and strong absorption of flux in the Lya forest. The reason is 
that this signature is imprinted by intrinsic and intervening neutral hydrogen, which is observed 
to be abundant in the early universe. The Lyman absorption signature thus provides a robust 
and sensitive means of identifying extremely high redshift galaxy candidates or of establishing an 
empirical limit on extremely high redshift galaxies using broad-band photometric techniques. 

Here we apply the Lyman absorption signature to search for galaxies at redshifts z 6 — 17 
using optical and infrared images of the HDF. Adopting plausible assumptions to relate rest-frame 
ultraviolet flux densities to unobscured star formation rates and adopting standard Friedmann 
cosmological models with dimensionless Hubble constant h = Hq/{100 km s^^ Mpc^^), deceleration 
parameter in the range qo = — 0.5, and zero cosmological constant, the infrared images are easily 
sensitive to the star formation rates expected for massive elliptical galaxy formation to quite high 
redshifts. For go = 0-5, the infrared images are sensitive to an unobscured star formation rate of 
M = 100 Mq yr~^ to redshifts as large as z = 17, and for qo = 0, the infrared images are 
sensitive to an unobscured star formation rate of M = 300 Mq yr"^ to redshifts as large as 
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z = 14. The primary result of the analysis is that only one extremely high redshift galaxy candidate 
is identified at the 5a level of significance (and four at the 4(T level), which implies a strict upper 
limit to the surface density of extremely high redshift galaxies and a strict upper limit to the volume 
density of extremely high redshift galaxies if (and only if) such galaxies are not highly obscured by 
dust. 

2. METHOD 

There are two reasons to believe that extremely high redshift galaxies must exhibit the same 
Lyman absorption signature that is characteristic of other high-redshift galaxies — namely complete 
absorption of flux below the Lyman limit and strong absorption of flux in the Lya forest. First, 
extremely high redshift galaxies are probably optically thick to Lyman continuum absorption by 
intrinsic neutral hydrogen. This is because extremely high redshift galaxies are probably charac- 
terized by much higher gas fractions than nearby galaxies, which are observed to be optically thick 
to Lyman continuum absorption by intrinsic neutral hydrogen (Leitherer et al. 1995). Second, the 
extremely high redshift universe is probably optically thick to Lyman continuum and Lya-forest 
absorption by intervening neutral hydrogen. This is because at the highest redshifts accessible to 
QSO absorption line measurements (z ~ 3 — 5), the universe is observed to be optically thick to Ly- 
man continuum and Lya-forest absorption, and the incidence of Lyman continuum and Lya-forest 
absorption systems is observed to be increasing rapidly with increasing redshift. For example, the 
Lya-forest flux decrement parameter Da is observed to increase monotonically with redshift to a 
value Da = 0.7 at z pa 5 (Madau 1995), and a straightforward linear extrapolation of measure- 
ments of Da over the redshift interval z w 2.5 — 5 predicts Da = 1 (i-e. complete absorption of 
flux in the Lyct forest) by z ~ 6.2. At even higher redshifts — at epochs before galaxy formation 
depletes the intergalactic medium and before ionizing photons produced by massive stars ionize 
the intergalactic medium — the universe is almost certainly optically thick to smoothly distributed 
Lyman continuum and Lya-forest Gunn-Peterson absorption. 

For these reasons, we conclude that the Lyman absorption signature provides a robust and sen- 
sitive means of identifying extremely high redshift galaxy candidates or of establishing an empirical 
limit on extremely high redshift galaxies using broad-band photometric techniques. The method 
is to apply the Lyman absorption signature to identify extremely high redshift galaxy candidates 
using optical and infrared photometry in just the same way that the Lyman absorption signature 
is applied to identify other high-redshift galaxy candidates using optical photometry alone, namely 
by seeking objects that exhibit a precipitous spectral discontinuity across Lya. At redshifts z > Q 
the Lyman absorption signature is redshifted beyond observed-frame optical wavelengths, hence 
extremely high redshift galaxy candidates must be identified as objects that are (1) detected at 
infrared wavelengths and (2) not detected to within sensitive upper limits at optical wavelengths. 

The major uncertainty in using photometric techniques to estimate redshifts of extremely high 
redshift galaxies is that owing to the lack of background probes the optical depth to Lya-forest 
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absorption has not yet been measured at redshifts beyond z ^ 5. Here we assume that at redshifts 

z > 6 the optical depth to Lya-forcst absorption approaches r — oo, which is consistent with the 
trend established at lower redshifts. In this case, at infrared wavelengths galaxies may in principle 
be detected at redshifts as large as 2; « 17, beyond which Lya is redshifted past the response of 
the K-band filter. 



3. ANALYSIS 

The optical images were obtained by HST in December, 1995 using the Wide Field Planetary 
Camera 2 (WFPC2) and the F300W, F450W, F606W, and F814W filters (Williams et al. 1996). 
The infrared images were obtained by the KPNO 4 m telescope in April, 1996 using the IRIM camera 

and standard J, H, and K filters (Dickinson et al. 1998, in preparation). Our previous analysis 
(Lanzctta, Yahil, & Fernandez-Soto 1996) identified objects in the F814W image, the faintest of 
which is of total magnitude ^5(8140) ~ 30. The goal of the current analysis is to identify objects 
in regions of the infrared images not occupied by objects detected at optical wavelengths. 

First, we measured the point spread functions of the infrared images by fitting Gaussian profiles 
to stars detected in the images. We accounted for variations of the point spread functions across 
the images by fitting the variations of the FWHM of the point spread functions versus positions 
from the centers of the images by quadratic polynomials. Next, we masked objects detected at 
optical wavelengths from the infrared images by setting the values of pixels associated with objects 
identified in the F814W image equal to zero. We located such pixels by modeling the spatial 
profile of every object detected in the F814W image as a convolution of the portion of the F814W 
image containing the object with the appropriate point spread function of the K-band image and 
fitting the model spatial profiles to the K-hand image to define isophotal mask regions. Next, we 
measured the noise characteristics of the infrared images by determining the empirical covariances 
of the masked images. The infrared images arc significantly correlated over adjacent pixels, hence 
off-diagonal terms of the empirical covariances must be included in order to determine accurate 
photometric uncertainties. Finally, we searched for objects detected in the infrared images by 
forming the "point source flux images" and corresponding "point source flux variance images" from 
the masked images. [We expect high-redshift galaxies to appear as point sources in the infrared 
images because our previous analysis (Lanzetta, Yahil, &i Fernandez-Soto et al. 1996) indicates that 
high-redshift galaxies are no more than several tenths arcsec across, which is much less than the 
FWHM ~ 1 arcsec point spread functions of the infrared images.] The optimally-weighted point 
source flux at pixel i,j is 

and the optimally- weighted point source flux variance at pixel i,j is 
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where f{i,j) is the flux at pixel a^{i,j) is the flux variance at pixel is the point 

spread function at pixel i,j, and A is a factor of order unity that accounts for off-diagonal terms 
of the empirical covariance and where the sums extend over all pixels covered by the point spread 
function. The point source flux images are shown in Figure 1, and the point source 5a detection 
thresholds determined from the point source flux variance images are listed in Table 1. Objects 
detected in the infrared images are identified by comparing pixels of the point source flux images 
with corresponding pixels of the point source variance images. The limiting magnitude thresholds 
are ^5(12,000) = 24.3, AS(16,000) = 23.7, and ^5(22,000) = 23.8. 

The result of the search is that one object at the 5a level of signiflcance and four objects at the 
Aa level of significance are detected in the K-hand image. We designate these objects as objects A 
through E in order of decreasing flux. No objects are detected in the J- or i7-band images at the 
5(7 level of significance. Properties of the objects are listed in Table 2, K-hand and F814W images 
of the objects are shown in Figure 2, and the spectral energy distributions of the objects are shown 
in Figure 3. The solid angle $7 of the portion of the infrared images included into the analysis is 
Q = 5.1 arcmin^, and the fraction / of the portion of the infrared images included into the analysis 
and associated with objects identified in the F814W image is / = 0.33. The solid angle ^1(1 — f) 
of the portion of the infrared images included into the analysis and not associated with objects 
identified in the F814W image is Q{1 — f) = 3.4 arcmin^. 

We performed several tests of the reality of objects A through E as follows: First, we sought to 
identify negative fiuctuations in the images employing exactly the same techniques used to identify 
positive fluctuations in the images. No negative fluctuations were detected in the J-, H-, or K- 
band images at the Aa or greater level of significance. Next, we estimated probabilities of detecting 
chance 4 or 5a fluctuations, assuming a Gaussian distribution of noise fluctuations. The portion 
of the infrared images included into the analysis is made up of ~ 4000 independent resolution 
elements, over which the probability of detecting a chance 5a fluctuation is just 1.2 x 10^*^ and of 
a chance Aa fluctuation is just 1.2 x 10^^. Finally, we sought to verify the objects in independent 
if -band images obtained by Hogg et al. (1996) with the Keck telescope. The Keck images (which 
cover only a small portion of the HDF) include only object D, which occurs at the very edge of 
the Keck image 2. Object D is present in the Keck image, with a flux consistent with the value 
measured from the KPNO image. We conclude that objects A through E are likely to be real rather 
than chance fluctuations. 



4. IMPLICATIONS 

The primary result of the analysis is that only one extremely high rcdshift galaxy candidate 
is identified at the 5a level of significance (and four at the Aa level), which implies strict upper 
limits to the surface and volume densities of extremely high redshift galaxies. Here we consider the 
5a level of significance threshold in order to establish conservative limits that are unlikely to be 
contaminated by spurious detections. 
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Following Steidel et al. (1996) and Lowenthal et al. (1997), we use the spectral synthesis models 
of Leitherer, Robert, & Heckman (1995) to relate rest-frame ultraviolet flux densities to unobscured 
star formation rates. We adopt a power-law relationship between monochromatic luminosity density 
Lx and wavelength A 

Lx{X) oc A-^ (3) 

and express unobscured star formation rate M in terms of monochromatic luminosity density -^^Aisoo 
at rest-frame wavelength A = 1500 A as 

M = ^^^isoo Mq yr-^ (4) 

1.4 X 1040 erg s"! A 

Equation (4) applies for a continuous star formation model with a Salpter (1955) initial mass 
function of upper mass cutoff M^p = 80Mq. A continuous star formation model is appropriate 
because the star formation time scale is expected to be no less than the dynamical time scale, which 
for massive elliptical galaxies is much longer than massive star lifetimes. We adopt (3 = 1.2, which 
is consistent with measurements of nearby starburst galaxies and galaxies with spectroscopically 
determined redshifts z ~ 3 in the HDF (Meurer et al. 1997) and with our own measurements of 
galaxies with photometrically estimated redshifts z ~ 5 in the HDF. For the adopted value of f3, 
the J- and i^-band images reach almost identical levels of sensitivity to star formation while the 
H-hand image reaches a lower level of sensitivity to star formation. For this reason, we consider 
only the K-hand image to identify extremely high redshift galaxies at all redshifts z 6 — 17. 

The unobscured star formation rate corresponding to the 5a detection threshold of the iiT-band 
image at redshifts 2; = 6 — 17 is shown in Figure 3. [For comparison, the unobscured star formation 
rate corresponding to the 5a detection threshold of the F814W image at redshifts z = — 6 
is also shown in Figure 3. This rate is calculated assuming a limiting magnitude threshold of 
AB (814:0) = 28.0.] For qq = 0.5, the infrared images are sensitive to an unobscured star formation 
rate of M = 30 h'"^ Mq yr"^ to redshifts as large as 2; = 12 and of M = 100 h'"^ Mq yr"^ to 
redshifts as large as 2; = 17. For go = 0, the infrared images are sensitive to an unobscured star 
formation rate of M = 100 Mq yr~^ to redshifts as large as z = 7 and of M = 300 h'"^ Mq 
-yT~^ to redshifts as large as 2; = 14. 

The result that only one extremely high redshift galaxy candidate is identified at the 5a level 
of significance establishes a 3a upper limit of 5.2 to the mean number of extremely high redshift 
galaxies over the area and volume sampled by the masked images. The solid angle sampled by 
the masked images is 3.4 arcmin"^, and the corresponding 3a upper limit to the surface density 
S of extremely high redshift galaxies to a limiting magnitude threshold AB(22,000) = 23.8 is 
S < 1.5 arcmin"^. The comoving volume sampled by the masked images between redshifts Zmm = 6 
and Zmax = 7 — 17 is listed in Table 3. For qo = 0.5, the corresponding 3a upper limit to the 
comoving volume density n of extremely high redshift galaxies with unobscured star formation 
rates exceeding M = 30 h~'^ Mq yr"^ is n < 1.8 x 10~^ Mpc^ and exceeding M = 100 h~'^ Mq 
yr~^ is n < 1.2 x 10~^ Mpc^. For qo = 0.5, the corresponding 3a upper limit to the comoving 
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volume density n of extremely high redshift galaxies with unobscured star formation rates exceeding 
M = 100 Mq yr-2 is n < 3.7 x 10""' Mpc^ and exceeding M = 300 h''^ Mq yr-^ is 
n < 3.1 X 10-5 Mpc^. 

5. DISCUSSION 

The 3a upper limits to the comoving volume density of extremely high redshift galaxies derived 
from analysis of the HDF images are substantially lower (by factors of between 3 and 180) than the 
volume density n = 5.6 x 10"^ Mpc~^ of nearby luminous {L > L^) galaxies (Ellis et al. 1996). 
This implies that either (1) early elliptical galaxies are obscured by dust, (2) intense star formation 
in early elliptical galaxies occurs with a short duty cycle, or (3) massive elliptical galaxies do not 
form at redshifts z > 6. Although these possibilities cannot be distinguished on the basis of the 
current observations, certain arguments can be brought to bear on their merits. 

First, it is presumably the case that the very first star formation in early elliptical galaxies 
is not obscured by dust, simply because the first dust is produced by the first generation of stars. 
Adopting the plausible assumption that star formation and the subsequent mixing of dust with gas 
cannot take place across an entire galaxy on a time scale that is much less than the dynamical time 
scale, then the dust-free phase of early galaxies could last for a significant fraction of the relevant 
djmamical time scale, which for massive elliptical galaxies is pa 10^ yr. Unless massive elliptical 
galaxy formation occurred at exceptionally high redshifts (i.e. z > 30), this phase of galaxy 
formation should persist through z < 17 and hence be in reach of direct observation, especially 
for cosmological models with large values of the deceleration parameter. Second, it is unlikely 
that intense star formation in early elliptical galaxies occurs on time scales much less than the 
dynamical time scale. Again, unless massive elliptical galaxy formation occurred at exceptionally 
high redshifts, this phase of galaxy formation should be in reach of direct observation. Although not 
conclusive, the evidence seems to indicate that massive elliptical galaxies did not form at redshifts 
z>6. 

If, in spite of the arguments put forward above, galaxies at the very early epochs probed by 
the infrared images of the HDF are obscured by even moderate amounts of dust, then of course 
substantially higher star formation rates are allowed. For example, recent evidence suggests that 
galaxies of redshift z ^ 3 suffer extinction of E{B — V)^ 0.1, which corresponds to an ultraviolet 
attenuation of a factor of several, depending on details of the extinction curve (e.g. Pettini et al. 
1997). If such a factor applies to the galaxies probed by the infrared images of the HDF, then star 
formation rates as large as pa 1000 Mq yr— 1 cannot be ruled out by the present observations. 

The upper limits to the comoving volume density of extremely high redshift galaxies derived 
from analysis of the HDF images are far more stringent — both in terms of sensitivity to star 
formation and of comoving volume sampled — than the upper limits derived from other searches for 
high-redshift galaxies at infrared wavelengths based on emission lines such as [O II] A3727 and H/3 
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(e.g. Thompson, Djorgovski, & Beckwith 1994). The reason is that the analysis of the HDF images 

is based on broad-band photometric techniques, which probe far greater depths than narrow-band 
techniques. The infrared images obtained by the KPNO 4 m telescope are not the deepest images 
that can be obtained with existing telescopes. We expect that the Lyman absorption signature 
applied to even deeper infrared images obtained with ground- or space-based telescopes will either 
identify a significant population of extremely high redshift galaxies or will establish a much more 
interesting empirical limit on extremely high redshift galaxies. 

Objects A through E are plausible candidates of galaxies at extremely high redshifts. Or they 
might be very cool stars of surface temperature T < 1400 K, although there is some indication 
that at least some of the objects are spatially resolved. These objects are prime targets for future 
observations. 

We are grateful to M. Dickinson for providing access to the reduced infrared images and to an 
anonymous referee for providing helpful comments. This research was supported by NASA grant 
NAGW-4422 and NSF grant AST-9624216. 
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TABLE 1 

POINT SOURCE 5cr DETECTION THRESHOLDS 







Detection Threshold 






fu 


fx 


Image 


AB 


(erg cm~^ Hz^^) 


(erg s""^ cm-2 A^^) 


J 


24.3 


6.8 X 10-3° 


1.4 X 10-19 


H 


23.7 


1.2 X 10-29 


1.4 X 10-19 


K 


. 23.8 


1.1 X 10-29 


6.8 X 10-20 
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TABLE 3 
COMOVING VOLUME 



V (Mpc^) 

^max go = go = 0.5 

7 14,020 609 

8 30,037 1150 

9 48,039 1633 

10 68,017 2069 

11 89,964 2464 

12 113,877 2825 

13 139,749 3157 

14 167,579 3462 

15 197,363 3745 

16 229,101 4008 

17 262,789 4253 
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Fig. 1. — Point source flux images determined from the (a) J-, (b) and (c) i^-band infrared 
images obtained by the KPNO 4 m telescope. Angular extent of each image is 163 x 163 arcsec^. 
Black represents high flux and white represents low flux. Solid gray areas indicate regions of the 
infrared images occupied by objects detected at optical wavelengths in the optical images obtained 
by HST. The angular extent of the infrared images is slightly larger than the angular extent of the 
optical images, hence no objects are detected at optical wavelengths along the edges of the infrared 
images or above and to the right of the location of the Planetary Camera (i.e. in the upper right 
corner). 

Fig. 2. — K-band and F814W images of objects A through E. Angular extent of each image is 
30 X 30 arcsGc^. Black represents high flux and white represents low flux. Tic marks indicate 
objects, i^-band images have been processed for optimal detection of point sources, as described 
by equation (1). 

Fig. 3. — Spectral energy distributions of objects A through E. Vertical error bars show flux 
uncertainties, and horizontal error bars show filter FWHM. 

Fig. 4. — Logarithm of unobscured star formation rate M vs. redshift z for (a) go = 0-5 and (b) 
go = 0. Star formation rate is calculated according to equations (3) and (4) assuming a limiting 
magnitude threshold of AB{22,000) = 23.8 at redshifts 2; = 6 — 17 and a limiting magnitude 
threshold of AB(8140) = 28.0 at redshifts ^ = - 6. 



